IRT/IRT as a newborn cystic fibrosis screening method: optimal cutoff points for a mixed population

Abstract: The Brazilian Unified National Health System (SUS) has incorporated newborn screening for cystic fibrosis since 2001. The protocol involves two samples of immunoreactive trypsinogen (IRT1/IRT2). This study aims to analyze fixed and floating values at the first and second IRT (IRT1/IRT2) cutoff points and assess the accuracy of the IRT/IRT methodology in a population from Northeastern Brazil. Descriptive, individual-level data from the newborn screening reference service data system (2013-2017) were used in this observational population study. The sensitivity, specificity, and positive predictive values (PPV) for the protocol were calculated. The best cutoff point was determined using the Youden’s index. The previous year’s cut-off values for the IRT1 and IRT2 99.4-, 99.5-, 99.6-, and 99.7-percentiles were utilized for the floating cutoff. During the studied period, 840,832 newborns underwent screening for cystic fibrosis, obtaining 49 cystic fibrosis diagnoses: 39 by newborn screening (79.6%) and 10 (20.4%) by clinical suspicion (false negative). The sensitivity, specificity, and PPV of the protocol totaled 79.6%, 99.9%, and 6.1%, respectively. No proposed cutoff for IRT1 performed better than the current one. IRT2 performed similarly to the current protocol at a cutoff point of 90ng/mL, showing the appropriate sensitivity and specificity while reducing the frequency of false positives. The protocol to screen newborns for cystic fibrosis had low sensitivity, a predictive positive value, and a high number of false positives and negatives. A floating cut point for IRT1 or IRT2 seems to constitute no viable option. However, changing the IRT2 cut point from 70ng/mL to 90ng/mL seems to have advantages and should undergo consideration.


Introduction
Newborn screening (NBS) provides the opportunity to diagnose diseases with clinical and economic importance to the health system in their early stages (preferably in their asymptomatic states) so early treatment may better benefit patients in the later stages of their diseases 1,2 .An accurate test or screening approach must be developed for a disease to support NBS.Ultimately, it involves balancing a sufficiently high sensitivity and an adequate specificity 2,3,4 .
Cystic fibrosis (CF), an inherited autosomal recessive disease with a chronic course and poor prognosis, is characterized by a defective gene that encodes the cystic fibrosis transmembrane conductance regulator (CFTR) protein.This condition unbalances water transport and sodium reabsorption, resulting in dysfunction in multiple organs 5,6 .
The incidence of cystic fibrosis varies greatly around the world.In Europe, it occurs in 1:2,500 live births, whereas the average incidence in North America totals 1:3,500 7 .According to the annual report of the Brazilian Group for Cystic Fibrosis Studies, based on the CF-NBS, the incidence of the disease in 2021 totaled 1/19,860 live births 8 .Many Brazilian states lack such data, especially those with the lowest development indicators, such as in Northern and Northeastern Brazil 9 .
The CF-NBS emerged when Crossley et al. 10 reported that immunoreactive trypsinogen (IRT), the precursor biomarker of pancreatic enzymes, increased in the blood of most newborns with CF due to abnormal enzymatic drainage, regardless of whether the individual had pancreatic insufficiency 10,11 .Currently, the initial stage of all NBS methods aims to determine IRT values.However, their second and third stages vary across geographic regions, depending on the incidence of the disease in the population, genetic background, intended targets, and the cost-effectiveness of the protocol, including saving resources by early diagnosis 3,11,12 .It should be remembered that NBS is not only a screening test but also a public health program with several benefits for the population and large costs to the government.Thus, evidence for its efficacy must support such budget.
The Brazilian Unified Naional Health System (SUS, acronym in Portuguese) incorporated NBS in 2001 by creating the Brazilian National Neonatal Screening Program, which covers all 26 states and the Federal District of Brazil 2,13 .One of the main objectives of the Brazilian National Neonatal Screening Program is to expand screening coverage to 100% of infants (or at least to 80%) 14 .Its original plan had three steps: first, to screen for phenylketonuria and congenital hypothyroidism; second, for sickle cell anemia and other hemoglobinopathies; and third, for cystic fibrosis.In 2014, the Brazilian National Neonatal Screening Program included two new diseases: congenital adrenal hyperplasia and biotinidase deficiency.The Brazilian National Neonatal Screening Program has recently added congenital toxoplasmosis to its protocol, investigating a total of seven diseases 15 .
Although the CF-NBS has been part of the Brazilian National Neonatal Screening Program since 2001, it is yet to be fully integrated in all states 16 .The Brazilian Ministry of Health has adopted the IRT/IRT protocol with the collection of up to two IRT samples at different time points.Briefly, primary health care units collect the first IRT sample (IRT1).A second sample should be collected if the IRT1 value exceeds 70ng/mL.A sweat test (ST) must be performed if the level of IRT2 exceeds 70ng/mL or if the IRT2 is collected after the first 30 days of the newborn's life.This screening IRT/IRT method is highly questionable because it depends on two samplings and has a low sensitivity and positive predictive value (PPV) 17,18 .
If the PPV of a screening test or a combined screening test is considered too low, it could be adjusted by raising or lowering the cutoff values for a continuous variable or by changing the components that make up a screening strategy 19 .Previous studies in the Brazilian Northeast and Southeast have shown that the IRT/IRT strategy has a low PPV and a high number of false positive cases 20,21 .As a result, recalling individuals and performing additional clinical and laboratory tests dramatically increases the cost of the CF-NBS in a continental nation like Brazil.
At a time when the Brazilian Ministry of Health is discussing the expansion of the national newborn screening program by including new diseases in screening 15 , it is important to improve the NBS strategies for the diseases that are already included in the Brazilian National Neonatal Screening Program and which still show room for improvement, such as CF.
Therefore, it is desirable that a study be conducted to determine the accuracy of the IRT/IRT protocol for CF-NBS in Brazil.Given the higher cost of using molecular techniques and the great genetic Cad.Saúde Pública 2024; 40 (7):e00150623 diversity 22,23,24,25 , especially in Northeastern Brazil, this independent analytical study of the current cutoff points in the IRT/IRT strategy aimed to analyze a few alternative fixed and floating IRT1 and IRT2 cutoff points.

Design and study period
Descriptive, individual-level data were retrospectively collected from 2013 to 2017 for this observational population study.

Population, site, and eligibility criteria
The Brazilian state of Bahia is located in Northeastern region.It has a population of approximately 15 million people.The studied years saw about 200,000 births per year.The state spans about 570,000km 2 and contains 417 municipalities.All newborns that were cared for by SUS within the NBS program of the Brazilian National Neonatal Screening Program in Bahia were eligible for this study.
Neonates for whom no new IRT1 or IRT2 sample were obtained after inadequate IRT1 or IRT2 sampling were excluded.Individuals from whom one sample was collected after 30 days of their lives, were unable to complete the IRT/IRT protocol, or died without diagnostic confirmation were also excluded.

Protocol for cystic fibrosis newborn screening in Bahia State (IRT/IRT)
The screening algorithm established in Bahia was in accordance with the recommendations of the Brazilian Group for Cystic Fibrosis Studies and the Brazilian Ministry of Health 2,26 .IRT was performed at up to two different time points (IRT/IRT protocol) as part of the strategy.Dry blood spots were collected on filter paper in about 4,500 primary healthcare units.These samples were delivered to a reference laboratory for newborn screening in the state capital, Salvador.The recommended range to collect IRT1 varied from the 3rd to the 5th days of life 2 .In cases of elevated IRT1 (≥ 70ng/mL), IRT2 was required and should ideally be collected from the 10th to the 21st days after birth.To collect a second sample of DBS from newborns with abnormal IRT1 levels, the NBS reference service contacted primary healthcare units.The sample was collected and delivered using the same procedure as the first IRT.If IRT2 was also elevated (≥ 70ng/mL), the infant should be referred to a CF care center for clinical evaluation and ST 2 .Bahia has two CF care centers, both of which are in the capital.Sampling for IRT1 or IRT2 dosages in individuals older than 30 days must be considered inadequate and unreliable 2,14 .

Processing of samples on filter paper
Blood samples were dried on filter paper and processed on AutoDELFIA (Waltham, United States) using the PerkinElmer Neonatal IRT kit (PerkinElmer do Brasil, São Paulo, Brazil) and an automated immunofluorometric method.

Sweat test
At the CF care centers, newborns were subjected to quantitative iontophoresis dosages using pilocarpine stimulation according to a predetermined algorithm.The sweat tests were interpreted as recommended 26,27 .In total, two positive tests at different times in several samples were required to confirm the disease 26 .

Study variables and implementation
The variables of interest were the annual number of live births in the state and the number of confirmed cases of CF by ST.IRT1 and IRT2 levels and the number of CF infants with negative CF-NBS were also described.
Samples were considered late if the IRT1 was collected after 15 days of life and inadequate if the IRT1 or IRT2 was collected after 30 days of life 2,26 .If the collected material was unable to be processed due to technical difficulties in collection, storage, or transport, the samples were also considered inadequate.

Source and preparation of data
All data on the number of examined newborns, IRT values, and time to IRT1 and IRT2 results were provided from the NBS Reference Service computerized information system.The number of live births in the State of Bahia was retrieved for each study year from the website of the Information Technology Department of the SUS (DASTASUS, acronym in Portuguese) 28 .To complete the database construction, the data were imported to Microsoft Excel for Mac, version 16.48 (https:// products.office.com/).

Data analysis
Stata, version 17.0 (https://www.stata.com);SPSS, version 21.0 (https://www.ibm.com/);Microsoft Excel for Mac, version 16.48; and OpenEpi (http://www.OpenEpi.com) 29were used for data tabulation and analysis.Statistical descriptions of data using means and/or medians with related dispersions (amplitude, variance, standard deviation, variation coefficient, and confidence interval) for quantitative indicators and simple and relative frequency measures for categories were employed.
Newborns were classified as false positives if they had an elevated IRT1 sample (≥ 70ng/mL) and a negative ST (< 30mEq/L) or both a high IRT1 and IRT2 (≥ 70ng/mL) and a negative ST. False negatives were defined as infants with a negative CF-NBS (IRT1 < 70ng/mL) or elevated IRT1 but a IRT2 < 70ng/mL, clinical findings consistent with CF, and a disease diagnosis confirmed by ST.
Sensitivity, specificity, PPV, negative predictive value (NPV), and overall accuracy with 95% confidence intervals (95%CI) were analyzed.In the first step, to assess accuracy, the CF-NBS protocol IRT1/IRT2 was considered as a single test, with ST serving as the gold standard.In the second step, the nonparametric DeLong method was used to determine the receiver operator characteristic curve (ROC) for the annual assessment of IRT1 and IRT2 cutoff points.Using the binomial approach, the area under the curve (AUC) was calculated with the corresponding 95%CI.Sensitivity, specificity, and number of false positives were evaluated for each IRT1 and IRT2 cutoff point.The optimal cutoff point was defined based on the highest sensitivity and specificity according to the Youden's index, which evaluates the ability of a diagnostic test to balance sensitivity (detection of disease) and specificity (detection of health or no disease).Therefore, the greatest Youden's index value associated with a given IRT score was offered as the optimal cutoff value for each year, using all data from the previous year (e.g., the 2013 IRT Youden's index was used as the 2014 IRT cutoff value).IRT1 and IRT2 values for the 99.4-, 99.5-, 99.6-, and 99.7-percentiles from the previous year were also evaluated to examine the influence of the sliding cutoff point.If the Youden's index IRT values were close in all examined 5 years, a fixed point within the found variation was used for an isolated analysis.Because of this and since other studies recommended 90ng/mL 18,30,31 , this value was used for IRT2.

Ethical concerns
The data in this research were retrieved as part of another study 21,32

Results
From 2013 to 2017, 1,017,576 live births occurred in Bahia, and 840,976 (82.6%) infants were subjected to CF-NBS.Of these, 6,640 (0.79%) IRT1 samples were classified as unsuitable for analysis.The second stage of CF-NBS included a 23.6% loss, totaling 3,510 infants subjected to IRT2 measurement (Figure 1).
During the study period, 49 infants were diagnosed with CF, 39 of whom were identified by NBS (79.6%).The remaining cases had CF-compatible symptoms and were subsequently confirmed as positive diagnoses, although screening results were initially negative (false negatives).A total of 13/39 (33.3%) children who tested positive for CF-NBS were correctly diagnosed in accordance with all steps of the diagnostic protocol (IRT1/IRT2/ST).Thus, the tests used to diagnose 26 infants consisted only of IRT1 and ST. Figure 1 shows an algorithm.A previous study has recently detailed these results 21 .
The current CF-NBS (IRT1/IRT2) could predict the presence of disease in 99.93% (95%CI: 99.92-99.93) of cases, with a 79.59% sensitivity (95%CI: 66.36-88.52)and a 100% specificity (95%CI: 100-100) from 2013 and 2017.Table 1 shows the annual and five-year accuracy statistics for the CF-NBS (IRT/IRT) and shows the number of valid IRT1 and IRT2 after excluding inappropriate samples and cases of death.The number of valid IRT2 tests was less than the number of false-positive cases.The observed AUC for IRT1 totaled 0.96 (95%CI: 0.9635-0.9643).The ROC curve for IRT2 evaded construction due to the small number of tested newborns (n = 13 in 5 years).Figure 2 shows these data and the operator characteristic curve (ROC) for all years (2013 to 2017).
IRT2 performance with floating cutoffs seems to be unapplicable.The use of the Youden's index indicated that the IRT2 cutoff point appears to be more effective at higher cutoff points, possibly reducing the proportion of false-positive results (Table 3).However, it was impossible to obtain these data for each year as only a few children underwent all CF-NBS steps (IRT1/IRT2).Nonetheless, when these measurements were available, the Youden's index found the ideal cutoff values to total 94.8ng/mL (2013), 116.0ng/mL (2014), 97.6ng/mL (2015), and 104ng/mL (2017).Thus, a fixed 90ng/ mL cutoff served to analyze the data, showing the same sensitivity and specificity as the current cutoff (70ng/mL) but with fewer false positives (Table 3).

Discussion
This study analyzed data from 840,976 infants subjected to CF-NBS from 2013 to 2017 in the State of Bahia.Of the 49 CF diagnosis, 39 were due to positive screenings and 10, to clinical suspicion − showing a negative CF-NBS (false negative).The accuracy of the screening protocol (IRT/IRT) for this specific population, with a 70ng/mL cutoff point, totaled 99.93%, with a 79.6% sensitivity, a 99.9% specificity, and a 6.1% PPV.When studying changes in cutoff values and the use of floating cutoff points of IRT2, this study found that a cutoff point near 90mg/mL seemed to provide the same sensitivity and specificity but with an improved PPV, resulting in a 46.7% (±10.2) decrease in the percent mean (standard deviation − SD) of false-positive results, with a 38.3% (2017) and 61.4% (2015) minimum and maximum reduction, respectively.
This study found a variation in sensitivity over time.Values were consistently lower than the minimum 95% recommended by the European Cystic Fibrosis Society (ECFS) 33 .This level of sensitivity suggests that the screening protocol used is scarcely successful in identifying individuals with the disease, thereby increasing the number of false-negative results.This study highlights the challenge of using this protocol to identify CF in newborns due to the high percentage of false-negative results (20.4%).Furthermore, this strategy showed a greater proportion of false-positive outcomes, which is associated with higher costs and a psychological burden for governments and families, respectively 18,34,35 .This is particularly relevant in states such as Bahia, which has considerable dis-Cad.Saúde Pública 2024; 40(7):e00150623 tances between most municipalities and the NBS service or CF care centers.Many other Brazilian states show the same reality.
Other studies have shown that the CF-NBS IRT/IRT protocol had a false-negative rate from 5 to 15% 20,36 .The genetic heterogeneity of the studied population may be a contributing factor to the higher number of false-negative results, which makes it difficult to determine the optimal cutoff points for this community 23,25,37,38 .A study conducted in Argentina evaluated the IRT/IRT protocol and found 14% of false-negative results with a 80% sensitivity.However, its sample size was much smaller than that in this study 31 .Studies conducted in Brazil, such as Maciel et al. 20 , also found a low 1.2% VPP with a false negative rate of 11.5%.Similar data have been found in Andalusia 17 and Argentina 31 .However, in all cases, the incidence of the disease exceeded that of the studied population, with rates of 1:6,675; 1:4,893 and 1:8,170 live births, respectively.Thus, the results of this study show good sensitivity and specificity (comparable to the other studies mentioned above) and a low PPV due to the lower incidence of the disease in our population.This highlights the importance of the methodological difficulties inherent in the test and the influence of epidemiologic conditions on the results of this study.The occurrence of false-negative outcome delays the CF diagnosis and may contribute to its worse prognosis 39 .
The low PPV values in this study are comparable to those in previous studies and seem to be related to the used methods 30,40 .ECFS guidelines recommend that the minimum acceptable PPV for CF-NBS should totaled 30% (0.3), significantly higher than the value in this study 33 .Nevertheless, it is important to note that the validity of these recommendations may be limited in communities with a significantly low incidence of CF.This is particularly relevant for the studied population, for which

Figure 2
Receiver operating characteristic (ROC) curve and area under the curve (AUC) for the cumulative first sample of immunoreactive trypsinogen (IRT1).
Area under the ROC curve = 0.9639.
Cad. Saúde Pública 2024; 40(7):e00150623 the incidence of the disease is estimated to total about 1:20,000 live births 21 .Therefore, despite the use of protocols with proven efficacy and high specificity, PPV tends to be consistently lower than the value the ECFS reported as it is also directly related to the incidence/prevalence of the disease 19 .Indeed, the high number of infants referred to ST despite being freed from the disease contributed to the increased costs of screening at SUS.Moreover, this situation placed a significant psychosocial burden on families until a definitive diagnosis was confirmed or ruled out 34 .This is primarily due to the low PPV associated with the screening process 7,35,41 .Even if the recommended PPV parameters are unsatisfied, decreasing the number of false-positive tests is a goal of the CF-NBS improvement initiative.
This study evaluated alternative cutoff points for IRT1 and the possibility of using floating cutoff points.However, the suggest that these options failed to be more effective than the current cutoff point.The low incidence of CF in the studied population may have acted as a negative determinant that made using floating cutoff points more difficult.When considering the decision to use brief time intervals for implementing the floating cut point approach, it is highly likely that CF was detected in a few infants, if any at all, making the calculation almost unattainable.
Although several studies suggest that the floating cutoff points increase testing efficiency and reduce the difficulties associated with IRT changes due to the heterogeneous genetic variants in populations, this study found otherwise 38,42,43 .IRT values significantly change with temperature, evident information in temperate regions for which the floating cutoff points seem more desirable 44,45 .The ineffectiveness of floating cutoff points may be attributable to the fact that this investigation was conducted in a tropical zone location with little change in annual temperature.
Based on the data obtained using the Youden's index as a method to choose the ideal IRT1 cutoff point for each year, it seemed that this value was significantly lower than the current fixed cutoff point in 2014 and 2017.It is worth considering whether these data indicate a realistic potential for lowering the IRT1 cutoff point to increase the sensitivity of the test by attempting to reduce the number of false-negative results.However, the performed analyses showed no evidence of improvement to the test sensitivity and specificity, suggesting that the change would unlikely increase the cost-effectiveness of CF-NBS.The best ratios between specificity and sensitivity for IRT1 in other years being very close to 70.0ng/mL confirm this result.
Lowering the IRT1 cutoff value effectively increases sensitivity and decreases the number of falsenegative results 19 .On the other hand, the number of false positives will rise, worsening the CF-NBS program and adding significantly to SUS and family expenditures 46,47 .In fact, municipalities such as Colorado and Buenos Aires, which adopted the same IRT/IRT methodology as this study, have implemented 50ng/mL IRT1 cutoff values, experiencing the anticipated effects 31,43 .he first stage of the procedures in a Belgian investigation used a similar 60ng/mL fixed IRT limit, but the protocol of that study employed molecular techniques 48 .Our data and analysis indicate that this approach seems inappropriate.If this change is considered, it is important to act prudently and weigh the risks and advantages of such alteration.
As previously mentioned, using the Youden's index to determine the ideal IRT2 cutoff point showed that, in three of the five assessed years, this number was higher than the one currently in use.Raising the IRT2 cutoff to 90ng/mL (fixed) preserves the sensitivity and specificity of the test and significantly decreases the incidence of false positives.These findings may suggest that the protocol of CF-NBS could be modified by increasing the IRT2 thresholds to 90 to 100ng/mL, which would likely have a positive impact.However, this datum must be viewed with great caution considering the small number of infants who underwent IRT2 measurement 18,49,50 .
There are other strategies to improve the accuracy of CF-NBS.Although the use of molecular methods for CF-NBS helps to increase the sensitivity of the test, it is important to recognize the limitations of its use in countries with limited resources, low incidence of the disease, and especially a high rate of CFTR gene heterogeneity.In 2003, Raskin et al. 24 showed high allele variability among Brazilian children with CF, making it difficult to import variant panels from other countries.In Bahia, a study showed that a panel including 20 variants was required to cover 90% of the CF infants in this population, detecting a previously undescribed mutation 38 .Moreover, allele variation among Brazilian states is large, so each locality would have to create its own CF gene panel 51 .Although the costs of molecular biology technologies have recently decreased, they remain steep, affecting the cost-Cad.Saúde Pública 2024; 40 (7):e00150623 effectiveness of this public health intervention and hindering its implementation in Brazilian states with limited government resources and lower human development indices.Thus, new strategies must be considered in this scenario.
Beyond examining variants in the CFTR gene, other screening strategies, such as pancreatitis-associated protein (PAP), should be considered as alternatives to alleviate the problems of the IRT/IRT strategy.The use of PAP associated with IRT without molecular technologies has shown an excellent cost-benefit ratio because it reuses the same dry blood stop sample that was collected during the initial test 31,46,48,52 , eliminating the need for a another blood draw in the primary care unit 52,53 .Several European countries have included the PAP approach with positive results in their screening protocols 30,48,54,55 .In Argentina, a comparison between IRT/IRT and IRT/PAP showed improved sensitivity and reduced number of false negatives 31 .However, that study was conducted with a small sample limited to the Buenos Aires population.Brazil requires further studies to determine whether the use of PAP is feasible.
It is important to mention the use of retrospective secondary data and screening-related losses as limitations of this study.Moreover, IRT2 analysis results should be considered with caution due to the small number of infants who completed all CF-NBS phases, with the IRT1/IRT2/ST procedures performed on a tiny minority of newborns.Because IRT2 sample collection occurred after 30 days of life in a large proportion of infants, only 53% of CF newborns underwent IRT1/ST.This explains the lower number of valid IRT2 tests than that of false positive tests.It should also be emphasized that the data refer only to the first 5 years of the NBS in Bahia and may not have been fully adequate for a proper evaluation of the accuracy of a complex and important public health program such as newborn screening.
In conclusion, this study indicates that the current CF-NBS protocol applied to the Bahia population has both low sensitivity and predictive positive value, with a significant number of false positives and negatives.The introduction of a floating cutoff point for IRT1 or IRT2 seems to offer an unviable option for the studied population.Changing the IRT2 cutoff point from 70ng/mL to values between 90ng/mL and 100ng/mL must have advantages and should be considered with caution.This measure fails to alter this routine and has no overall cost to SUS.A reduction in the IRT1 threshold has been proposed.However, the results of this study preclude the precise definition of a value.Any such adjustment should be carefully considered as it may do more harm than good.Implementation of new strategies with PAP or CFTR DNA analysis requires a pilot study to confirm this hypothesis.
Although far from being ideal, the current strategy is successful and should be maintained as a CF-NBS since promoting and encouraging NBS evaluation programs are crucial steps to continuously improve public policies, which must be done often.Further studies must be conducted to evaluate screening methods to selecting the method with the best cost-benefit ratio considering local disease incidence and the genetic and phenotypic characteristics of its population.

Figure 1
Figure 1Results of screening for cystic fibrosis in newborns in Bahia State, Brazil (2013-2017).

Fibrosis
Quística; Tamizaje Neonatal; Exactitud de los Datos; Programas de Detección Diagnóstica; Programas Naionales de Salud Submitted on 16/Aug/2023 Final version resubmited on 27/Mar/2024 Approved on 01/Apr/2024 (CAAE: 79495717.1.0000.0049),which was subjected to the Research Ethics Committee of University Hospital Professor Edgard Santos and approved by it under number 2.389.852/2017.This study is in accordance with the research ethics principles of the Declaration of Helsinki and Resolution n. 466/2012 of the Brazilian National Health Council.

Table 1
Annual and cumulative accuracy assessment of the protocol to newborns screening for cystic fibrosis (IRT/IRT).

Table 2
Annual accuracy data for the first sample of immunoreactive trypsinogen (IRT1) at distinct cutoff points

Table 3
Annual accuracy data for the second sample of immunoreactive trypsinogen (IRT2) at distinct cutoff points.